Application and Evaluation of a Revision Framework to the Formative Assessments in a MOOC

In a MOOC development process, it is important to analyze learner data after completion of the course as a feedback mechanism for revision in future iterations of the course. The cyclic nature of incorporating data analysis into course development aligns with the process of learning engineering. Many course designers and instructors struggle to find an easy method to evaluate formative assessments systematically to inform revision approaches. In this study, we applied a framework for the revision of formative assessments to identify the source, the cause, and a revision strategy for low-performance questions in the quizzes from multiple runs of a MOOC. The revisions led to an improvement in the percentage of learners correctly answering a subset of those impacted questions with significance, while all other percentages had no change of significance. These score improvements were not due to the reuse of the same questions in multiple runs of a MOOC. We concluded that the framework effectively applied to this online course and was impactful in resulting in positive changes. These results encourage others to try the revision process in their courses in the future.